{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recommender Systems 2020/21\n",
    "\n",
    "## Practice session on MF recommenders.\n",
    "\n",
    "### Outline\n",
    "* MF Recommenders\n",
    "* BPR-MF\n",
    "* PureSVD\n",
    "* Comparison between MF recommenders\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Administrative code\n",
    "\n",
    "Download the dataset and generate _TRAIN_ and _TEST_ splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movielens10M: Verifying data consistency...\n",
      "Movielens10M: Verifying data consistency... Passed!\n",
      "DataReader: current dataset is: <class 'Data_manager.Dataset.Dataset'>\n",
      "\tNumber of items: 10681\n",
      "\tNumber of users: 69878\n",
      "\tNumber of interactions in URM_all: 10000054\n",
      "\tValue range in URM_all: 0.50-5.00\n",
      "\tInteraction density: 1.34E-02\n",
      "\tInteractions per user:\n",
      "\t\t Min: 2.00E+01\n",
      "\t\t Avg: 1.43E+02\n",
      "\t\t Max: 7.36E+03\n",
      "\tInteractions per item:\n",
      "\t\t Min: 0.00E+00\n",
      "\t\t Avg: 9.36E+02\n",
      "\t\t Max: 3.49E+04\n",
      "\tGini Index: 0.57\n",
      "\n",
      "\tICM name: ICM_genres, Value range: 1.00 / 1.00, Num features: 20, feature occurrences: 21564, density 1.01E-01\n",
      "\tICM name: ICM_tags, Value range: 1.00 / 69.00, Num features: 10217, feature occurrences: 108563, density 9.95E-04\n",
      "\tICM name: ICM_all, Value range: 1.00 / 69.00, Num features: 10237, feature occurrences: 130127, density 1.19E-03\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from Notebooks_utils.data_splitter import train_test_holdout\n",
    "from Data_manager.Movielens.Movielens10MReader import Movielens10MReader\n",
    "\n",
    "data_reader = Movielens10MReader()\n",
    "datasets_dict = data_reader.load_data()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 0)\t5.0\n",
      "  (0, 1)\t5.0\n",
      "  (0, 2)\t5.0\n",
      "  (0, 3)\t5.0\n",
      "  (0, 4)\t5.0\n",
      "  (0, 5)\t5.0\n",
      "  (0, 6)\t5.0\n",
      "  (0, 7)\t5.0\n",
      "  (0, 8)\t5.0\n",
      "  (0, 9)\t5.0\n",
      "  (0, 10)\t5.0\n",
      "  (0, 11)\t5.0\n",
      "  (0, 12)\t5.0\n",
      "  (0, 13)\t5.0\n",
      "  (0, 14)\t5.0\n",
      "  (0, 15)\t5.0\n",
      "  (0, 16)\t5.0\n",
      "  (0, 17)\t5.0\n",
      "  (0, 18)\t5.0\n",
      "  (0, 19)\t5.0\n",
      "  (0, 20)\t5.0\n",
      "  (0, 21)\t5.0\n",
      "  (1, 16)\t3.0\n",
      "  (1, 22)\t5.0\n",
      "  (1, 23)\t3.0\n",
      "  :\t:\n",
      "  (69877, 463)\t3.0\n",
      "  (69877, 467)\t1.0\n",
      "  (69877, 468)\t4.0\n",
      "  (69877, 475)\t2.0\n",
      "  (69877, 481)\t3.0\n",
      "  (69877, 486)\t4.0\n",
      "  (69877, 505)\t3.0\n",
      "  (69877, 518)\t1.0\n",
      "  (69877, 537)\t5.0\n",
      "  (69877, 541)\t2.0\n",
      "  (69877, 1081)\t2.0\n",
      "  (69877, 1302)\t4.0\n",
      "  (69877, 1322)\t2.0\n",
      "  (69877, 1436)\t4.0\n",
      "  (69877, 1609)\t1.0\n",
      "  (69877, 1646)\t3.0\n",
      "  (69877, 1660)\t2.0\n",
      "  (69877, 1671)\t2.0\n",
      "  (69877, 2001)\t4.0\n",
      "  (69877, 2065)\t1.0\n",
      "  (69877, 2941)\t1.0\n",
      "  (69877, 3066)\t1.0\n",
      "  (69877, 3386)\t3.0\n",
      "  (69877, 3448)\t1.0\n",
      "  (69877, 5330)\t1.0\n"
     ]
    }
   ],
   "source": [
    "URM_all = datasets_dict.AVAILABLE_URM[\"URM_all\"]\n",
    "print(URM_all)\n",
    "\n",
    "URM_train, URM_test = train_test_holdout(URM_all, train_perc = 0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MF Computing prediction\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/988/1*tiF4e4Y-wVH732_6TbJVmQ.png\" alt=\"Latent Factors\" style=\"margin: 0 auto;\"/>\n",
    "\n",
    "In a MF model you have two matrices, let's call them $U$ and $V$. \n",
    "\n",
    "In $U$ you have one row for every user (i.e., users are in the rows). In $V$ you have one column for every item (i.e., items are in the columns). The other dimension (columns for $U$ and rows for $V$) is called latent factors.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://miro.medium.com/max/988/1*tiF4e4Y-wVH732_6TbJVmQ.png\" alt=\"Latent Factors\" style=\"margin: 0 auto;\"/>\n",
    "\n",
    "\n",
    "The number of latent factors is variable (a hyperparameter), if we represent the number of item factors by $d$, then $U$ is an $m \\times d$ matrix and $V$ is a $d \\times n$ matrix. Lastly, $U$ is called the user factors matrix and $V$ is the item factors matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 69878, 10681)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_factors = 10\n",
    "num_users, num_items = URM_train.shape\n",
    "\n",
    "num_factors, num_users, num_items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# user_factors is our U, U is n_users x num_factors\n",
    "user_factors = np.random.random((num_users, num_factors))\n",
    "\n",
    "# item_factors is our V\n",
    "item_factors = np.random.random((num_items, num_factors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[0.77105129, 0.26981857, 0.21525026, ..., 0.81810914, 0.18029247,\n",
       "         0.97124718],\n",
       "        [0.73256688, 0.19107128, 0.0166121 , ..., 0.93479036, 0.27351954,\n",
       "         0.82409332],\n",
       "        [0.01614075, 0.59567967, 0.00828774, ..., 0.00915704, 0.81327579,\n",
       "         0.73391817],\n",
       "        ...,\n",
       "        [0.50609718, 0.51421094, 0.53414818, ..., 0.19367252, 0.89929362,\n",
       "         0.72968165],\n",
       "        [0.66306608, 0.50086656, 0.2686102 , ..., 0.71329147, 0.98767928,\n",
       "         0.09459023],\n",
       "        [0.38507525, 0.95404299, 0.62014664, ..., 0.67315072, 0.02186156,\n",
       "         0.35256418]]),\n",
       " (69878, 10))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_factors, user_factors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[0.34736724, 0.15438443, 0.68194495, ..., 0.16829192, 0.31393316,\n",
       "         0.32430294],\n",
       "        [0.68942466, 0.78205893, 0.10659545, ..., 0.3056192 , 0.48205477,\n",
       "         0.12945008],\n",
       "        [0.20601237, 0.33132837, 0.58948576, ..., 0.73873408, 0.73419341,\n",
       "         0.63150557],\n",
       "        ...,\n",
       "        [0.17463286, 0.36053267, 0.77677085, ..., 0.93123106, 0.55821074,\n",
       "         0.8863073 ],\n",
       "        [0.65169201, 0.71092907, 0.32342315, ..., 0.67654419, 0.33397039,\n",
       "         0.68935113],\n",
       "        [0.9926887 , 0.13838396, 0.63449694, ..., 0.98416582, 0.88046898,\n",
       "         0.10438477]]),\n",
       " (10681, 10))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_factors, item_factors.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To compute the prediction we have to muliply the user factors to the item factors\n",
    "\n",
    "$$\\hat{x}_{ui} = \\langle w_u,h_i \\rangle = \\sum_{f=1}^k w_{uf} \\cdot h_{if}$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Zooming-in in an specific user.\n",
    "\n",
    "Let's suppose Bob is user 42.\n",
    "\n",
    "<img src=\"images/bob_example.jpeg\" style=\"margin: 0 auto;width: 250px;\" alt=\"Bob Image\">\n",
    "\n",
    "And that we're interested into the movie Toy Story (with id 15).\n",
    "\n",
    "<img src=\"images/toy_story_poster.png\" style=\"margin: 0 auto;width: 250px;\" alt=\"Toy Story Poster Image\">\n",
    "\n",
    "In our specific case, let's see what is the rating for user $u = 42$ and item $i = 15$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction is 1.51\n"
     ]
    }
   ],
   "source": [
    "item_index = 15\n",
    "user_index = 42\n",
    "\n",
    "prediction = np.dot(user_factors[user_index,:], item_factors[item_index,:])\n",
    "\n",
    "print(\"Prediction is {:.2f}\".format(prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MF BPR models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Recap on BPR\n",
    "S.Rendle et al. BPR: Bayesian Personalized Ranking from Implicit Feedback. UAI2009\n",
    "\n",
    "The usual approach for item recommenders is to predict a personalized score $\\hat{x}_{ui}$ for an item that reflects the preference of the user for the item. Then the items are ranked by sorting them according to that score.\n",
    "\n",
    "Machine learning approaches are tipically fit by using observed items as a positive sample and missing ones for the negative class. A perfect model would thus be useless, as it would classify as negative (non-interesting) all the items that were non-observed at training time. The only reason why such methods work is regularization.\n",
    "\n",
    "BPR use a different approach. The training dataset is composed by triplets $(u,i,j)$ representing that user u is assumed to prefer i over j. For an implicit dataset this means that u observed i but not j:\n",
    "$$D_S := \\{(u,i,j) \\mid i \\in I_u^+ \\wedge j \\in I \\setminus I_u^+\\}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### BPR-OPT\n",
    "A machine learning model can be represented by a parameter vector $\\Theta$ which is found at fitting time. BPR wants to find the parameter vector that is most probable given the desired, but latent, preference structure $>_u$:\n",
    "$$p(\\Theta \\mid >_u) \\propto p(>_u \\mid \\Theta)p(\\Theta) $$\n",
    "$$\\prod_{u\\in U} p(>_u \\mid \\Theta) = \\dots = \\prod_{(u,i,j) \\in D_S} p(i >_u j \\mid \\Theta) $$\n",
    "\n",
    "The probability that a user really prefers item $i$ to item $j$ is defined as:\n",
    "$$ p(i >_u j \\mid \\Theta) := \\sigma(\\hat{x}_{uij}(\\Theta)) $$\n",
    "Where $\\sigma$ represent the logistic sigmoid and $\\hat{x}_{uij}(\\Theta)$ is an arbitrary real-valued function of $\\Theta$ (the output of your arbitrary model).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "To complete the Bayesian setting, we define a prior density for the parameters:\n",
    "$$p(\\Theta) \\sim N(0, \\Sigma_\\Theta)$$\n",
    "And we can now formulate the maximum posterior estimator:\n",
    "$$BPR-OPT := \\log p(\\Theta \\mid >_u) $$\n",
    "$$ = \\log p(>_u \\mid \\Theta) p(\\Theta) $$\n",
    "$$ = \\log \\prod_{(u,i,j) \\in D_S} \\sigma(\\hat{x}_{uij})p(\\Theta) $$\n",
    "$$ = \\sum_{(u,i,j) \\in D_S} \\log \\sigma(\\hat{x}_{uij}) + \\log p(\\Theta) $$\n",
    "$$ = \\sum_{(u,i,j) \\in D_S} \\log \\sigma(\\hat{x}_{uij}) - \\lambda_\\Theta ||\\Theta||^2 $$\n",
    "\n",
    "Where $\\lambda_\\Theta$ are model specific regularization parameters.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### BPR learning algorithm\n",
    "Once obtained the log-likelihood, we need to maximize it in order to find our obtimal $\\Theta$. As the criterion is differentiable, gradient descent algorithms are an obvious choiche for maximization.\n",
    "\n",
    "Gradient descent comes in many fashions, you can find an overview on this thesis https://www.politesi.polimi.it/bitstream/10589/133864/3/tesi.pdf on pages 18-19-20. A nice post about momentum is available here https://distill.pub/2017/momentum/\n",
    "\n",
    "The basic version of gradient descent consists in evaluating the gradient using all the available samples and then perform a single update. The problem with this is, in our case, that our training dataset is very skewed. Suppose an item $i$ is very popular. Then we have many terms of the form $\\hat{x}_{uij}$ in the loss because for many users u the item $i$ is compared against all negative items $j$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The other popular approach is stochastic gradient descent, where for each training sample an update is performed. This is a better approach, but the order in which the samples are traversed is crucial. To solve this issue BPR uses a stochastic gradient descent algorithm that choses the triples randomly.\n",
    "\n",
    "The gradient of BPR-OPT with respect to the model parameters is: \n",
    "$$\\frac{\\partial BPR-OPT}{\\partial \\Theta} = \\sum_{(u,i,j) \\in D_S} \\frac{\\partial}{\\partial \\Theta} \\log \\sigma (\\hat{x}_{uij}) - \\lambda_\\Theta \\frac{\\partial}{\\partial\\Theta} || \\Theta ||^2$$\n",
    "$$ =  \\sum_{(u,i,j) \\in D_S} \\frac{-e^{-\\hat{x}_{uij}}}{1+e^{-\\hat{x}_{uij}}} \\frac{\\partial}{\\partial \\Theta}\\hat{x}_{uij} - \\lambda_\\Theta \\Theta $$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### BPR-MF\n",
    "\n",
    "In order to practically apply this learning schema to an existing algorithm, we first split the real valued preference term: $\\hat{x}_{uij} := \\hat{x}_{ui} − \\hat{x}_{uj}$. And now we can apply any standard collaborative filtering model that predicts $\\hat{x}_{ui}$.\n",
    "\n",
    "The problem of predicting $\\hat{x}_{ui}$ can be seen as the task of estimating a matrix $X:U×I$. With matrix factorization the target matrix $X$ is approximated by the matrix product of two low-rank matrices $W:|U|\\times k$ and $H:|I|\\times k$:\n",
    "$$X := WH^t$$\n",
    "The prediction formula can also be written as:\n",
    "$$\\hat{x}_{ui} = \\langle w_u,h_i \\rangle = \\sum_{f=1}^k w_{uf} \\cdot h_{if}$$\n",
    "Besides the dot product ⟨⋅,⋅⟩, in general any kernel can be used.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can now specify the derivatives:\n",
    "$$ \\frac{\\partial}{\\partial \\theta} \\hat{x}_{uij} = \\begin{cases}\n",
    "(h_{if} - h_{jf}) \\text{ if } \\theta=w_{uf}, \\\\\n",
    "w_{uf} \\text{ if } \\theta = h_{if}, \\\\\n",
    "-w_{uf} \\text{ if } \\theta = h_{jf}, \\\\\n",
    "0 \\text{ else }\n",
    "\\end{cases} $$\n",
    "\n",
    "Which basically means: user $u$ prefer $i$ over $j$, let's do the following:\n",
    "- Increase the relevance (according to $u$) of features belonging to $i$ but not to $j$ and vice-versa\n",
    "- Increase the relevance of features assigned to $i$\n",
    "- Decrease the relevance of features assigned to $j$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Train a MF MSE model\n",
    "\n",
    "### Use SGD as we saw for SLIM\n",
    "\n",
    "Keeping the same idea for $u = 42$ and $i = 15$, let's suppose that the real rating $r_{u,i} = 5$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Prediction error is 3.49\n",
      "* Real value (r_u,i) is 5\n",
      "* Prediction (rhat_u,i) is 1.51\n"
     ]
    }
   ],
   "source": [
    "test_data = 5\n",
    "\n",
    "prediction = np.dot(user_factors[user_index,:], item_factors[item_index,:])\n",
    "gradient = test_data - prediction\n",
    "\n",
    "print(f\"* Prediction error is {gradient:.2f}\")\n",
    "print(f\"* Real value (r_u,i) is {test_data}\")\n",
    "print(f\"* Prediction (rhat_u,i) is {prediction:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Remember, for SGD we have a regularization parameter $\\lambda_\\Theta$ and the learning rate hyperparameter $lr$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "learning_rate = 1e-2\n",
    "regularization = 1e-3\n",
    "\n",
    "# Copy original value to avoid messing up the updates\n",
    "H_i = item_factors[item_index,:]\n",
    "W_u = user_factors[user_index,:]\n",
    "\n",
    "user_factors[user_index,:] += learning_rate * (gradient * H_i - regularization * W_u)\n",
    "item_factors[item_index,:] += learning_rate * (gradient * W_u - regularization * H_i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now that we updated the user and item factors, let's see the new prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Prediction error is 3.30\n",
      "* Real value (r_u,i) is 5\n",
      "* Prediction (rhat_u,i) is 1.70\n"
     ]
    }
   ],
   "source": [
    "new_prediction = np.dot(user_factors[user_index,:], item_factors[item_index,:])\n",
    "new_gradient = test_data - new_prediction\n",
    " \n",
    "print(f\"* Prediction error is {new_gradient:.2f}\")\n",
    "print(f\"* Real value (r_u,i) is {test_data}\")\n",
    "print(f\"* Prediction (rhat_u,i) is {new_prediction:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Just for comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature\t\t\t|Previous Values \t|After Gradient values\n",
      "___________________________________________________________________________\n",
      "Prediction Error\t|3.49\t\t\t|3.30\n",
      "Real value (r_u,i)\t|5\t\t\t|5\n",
      "Prediction (rhat_u,i)\t|1.51\t\t\t|1.70\n"
     ]
    }
   ],
   "source": [
    "print(f\"Feature\\t\\t\\t|Previous Values \\t|After Gradient values\")\n",
    "print(f\"___________________________________________________________________________\")\n",
    "print(f\"Prediction Error\\t|{gradient:.2f}\\t\\t\\t|{new_gradient:.2f}\")\n",
    "print(f\"Real value (r_u,i)\\t|{test_data}\\t\\t\\t|{test_data}\")\n",
    "print(f\"Prediction (rhat_u,i)\\t|{prediction:.2f}\\t\\t\\t|{new_prediction:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ⚠️WARNING: Initialization must be done with random non-zero values⚠️\n",
    "\n",
    "... otherwise this happens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "user_factors = np.zeros((num_users, num_factors))\n",
    "item_factors = np.zeros((num_items, num_factors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Prediction error is 5.00\n",
      "* Real value (r_u,i) is 5\n",
      "* Prediction (rhat_u,i) is 0.0\n"
     ]
    }
   ],
   "source": [
    "prediction = np.dot(user_factors[user_index,:], item_factors[item_index,:])\n",
    "gradient = test_data - prediction\n",
    "\n",
    "print(f\"* Prediction error is {gradient:.2f}\")\n",
    "print(f\"* Real value (r_u,i) is {test_data}\")\n",
    "print(f\"* Prediction (rhat_u,i) is {prediction}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "W_u = user_factors[user_index,:]\n",
    "H_i = item_factors[item_index,:]\n",
    "\n",
    "user_factors[user_index,:] += learning_rate * (gradient * H_i - regularization * W_u)\n",
    "item_factors[item_index,:] += learning_rate * (gradient * W_u - regularization * H_i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Prediction error is 5.00\n",
      "* Real value (r_u,i) is 5\n",
      "* Prediction (rhat_u,i) is 0.0\n"
     ]
    }
   ],
   "source": [
    "new_prediction = np.dot(user_factors[user_index,:], item_factors[item_index,:])\n",
    "new_gradient = test_data - new_prediction\n",
    "\n",
    "print(f\"* Prediction error is {new_gradient:.2f}\")\n",
    "print(f\"* Real value (r_u,i) is {test_data}\")\n",
    "print(f\"* Prediction (rhat_u,i) is {new_prediction}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Since the updates multiply the gradient and the latent factors, if those are zero the SGD will never be able to move from that point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Now with BPR MF\n",
    "\n",
    "The basics are the same, except for how we compute the gradient, where we have to sample triplets $(u, i^+, i^-)$ instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(69878, 69802)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "URM_mask = URM_train.copy()\n",
    "URM_mask.data[URM_mask.data <= 3] = 0\n",
    "\n",
    "URM_mask.eliminate_zeros()\n",
    "\n",
    "# Extract users having at least one interaction to choose from\n",
    "eligibleUsers = []\n",
    "\n",
    "for user_id in range(num_users):\n",
    "    start_pos = URM_mask.indptr[user_id]\n",
    "    end_pos = URM_mask.indptr[user_id+1]\n",
    "\n",
    "    if len(URM_mask.indices[start_pos:end_pos]) > 0:\n",
    "        eligibleUsers.append(user_id)  \n",
    "        \n",
    "num_users, len(eligibleUsers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def sampleTriplet():\n",
    "    # By randomly selecting a user in this way we could end up \n",
    "    # with a user with no interactions\n",
    "    #user_id = np.random.randint(0, n_users)\n",
    "    user_id = np.random.choice(eligibleUsers)\n",
    "    \n",
    "    # Get user seen items and choose one\n",
    "    userSeenItems = URM_mask[user_id,:].indices\n",
    "    pos_item_id = np.random.choice(userSeenItems)\n",
    "\n",
    "    negItemSelected = False\n",
    "\n",
    "    # It's faster to just try again than to build a mapping of the non-seen items\n",
    "    while (not negItemSelected):\n",
    "        neg_item_id = np.random.randint(0, num_items)\n",
    "\n",
    "        if (neg_item_id not in userSeenItems):\n",
    "            negItemSelected = True\n",
    "\n",
    "    return user_id, pos_item_id, neg_item_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(61434, 1059, 3727)\n",
      "(57395, 266, 5167)\n",
      "(29218, 1371, 3484)\n",
      "(6548, 137, 1429)\n",
      "(51378, 391, 4220)\n",
      "(21840, 584, 4551)\n",
      "(1986, 1089, 543)\n",
      "(62562, 1611, 3028)\n",
      "(58025, 327, 7243)\n",
      "(2145, 1077, 5364)\n"
     ]
    }
   ],
   "source": [
    "for _ in range(10):\n",
    "    print(sampleTriplet())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "user_factors = np.random.random((num_users, num_factors))\n",
    "item_factors = np.random.random((num_items, num_factors))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2518 864 145\n"
     ]
    }
   ],
   "source": [
    "user_id, positive_item, negative_item = sampleTriplet()\n",
    "\n",
    "print(user_id, positive_item, negative_item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0.25018247, 0.45896203, 0.66343316, 0.1443068 , 0.27183938,\n",
       "        0.56821441, 0.39703548, 0.44701106, 0.15537737, 0.3435701 ]),\n",
       " array([0.03223317, 0.80434417, 0.89772161, 0.81113855, 0.4956003 ,\n",
       "        0.85430307, 0.64105046, 0.34943407, 0.08350337, 0.42093353]),\n",
       " array([0.51465736, 0.85777222, 0.33279313, 0.58213778, 0.51465036,\n",
       "        0.87294734, 0.49067511, 0.92394343, 0.96496654, 0.26542339]))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_factors[user_id, :], item_factors[positive_item,:], item_factors[negative_item,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.03378784701867701"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_uij = np.dot(user_factors[user_id, :], \n",
    "               (item_factors[positive_item,:] - item_factors[negative_item,:]))\n",
    "x_uij "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5084461582456739"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid_item = 1 / (1 + np.exp(x_uij))\n",
    "sigmoid_item"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### When using BPR we have to update three components, the user factors and the item factors of both the positive and negative item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "H_i = item_factors[positive_item,:]\n",
    "H_j = item_factors[negative_item,:]\n",
    "W_u = user_factors[user_id,:]\n",
    "\n",
    "user_factors[user_index,:] += learning_rate * (sigmoid_item * ( H_i - H_j ) - regularization * W_u)\n",
    "item_factors[positive_item,:] += learning_rate * (sigmoid_item * ( W_u ) - regularization * H_i)\n",
    "item_factors[negative_item,:] += learning_rate * (sigmoid_item * (-W_u ) - regularization * H_j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.017205978726479706"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_x_uij = np.dot(user_factors[user_id, :], (item_factors[positive_item,:] - item_factors[negative_item,:]))\n",
    "new_x_uij"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature\t\t\t|Previous Values \t|After Gradient values\n",
      "___________________________________________________________________________\n",
      "X_uij\t\t\t|-0.03379\t\t|-0.01721\n"
     ]
    }
   ],
   "source": [
    "print(f\"Feature\\t\\t\\t|Previous Values \\t|After Gradient values\")\n",
    "print(f\"___________________________________________________________________________\")\n",
    "print(f\"X_uij\\t\\t\\t|{x_uij:.5f}\\t\\t|{new_x_uij:.5f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### How to rank items with MF ?\n",
    "\n",
    "Compute the prediction for all items and rank them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.88613433, 3.26810738, 2.23810243, ..., 2.97465278, 2.98534907,\n",
       "       2.7113013 ])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_scores = np.dot(user_factors[user_index,:], item_factors.T)\n",
    "item_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10681,)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_scores.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Early stopping, how to use it and when it is needed\n",
    "\n",
    "Problem, how many epochs? 5, 10, 150, 2487 ?\n",
    "\n",
    "We could try different values in increasing order: 5, 10, 15, 20, 25...\n",
    "\n",
    "### However, in this way we would train up to a point, test and then discard the model, to re-train it again up to that same point and then some more... not a good idea.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Early stopping! \n",
    "* Train the model up to a certain number of epochs, say 5\n",
    "* Compute the recommendation quality on the validation set\n",
    "* Train for other 5 epochs\n",
    "* Compute the recommendation quality on the validation set AND compare it with the previous one. If better, then we have another best model, if not, go ahead...\n",
    "* Repeat until you have either reached the max number of epoch you want to allow (e.g., 300) or a certain number of contiguous validation seps have not updated te best model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Advantages:\n",
    "* Easy to implement, we already have all that is required, a train function, a predictor function and an evaluator\n",
    "* MUCH faster than retraining everything from the beginning\n",
    "* Often allows to reach even better solutions\n",
    "\n",
    "### Challenges:\n",
    "* The evaluation step may be very slow compared to the time it takes to re-train the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## PureSVD model\n",
    "\n",
    "### As opposed to the previous ones, PureSVD relies on the SVD decomposition of the URM, which is an easily available function\n",
    "\n",
    "In our case, an SVD decomposition of the URM ($m \\times n$)is as follows\n",
    "\n",
    "$$ URM = U \\Sigma V^T $$\n",
    "\n",
    "Where $U$ is an orthogonal $m \\times m$ matrix, $\\Sigma$ is a rectangular diagonal matrix ($m \\times n$), and $V^T$ is an orthogonal $n \\times n$ matrix. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "However, calculating the SVD for the whole URM consumes lots of resources (time and memory), we can use a *truncated* version of SVD called *Truncated SVD*, the idea is similar to the original SVD, but instead of calculating an *exact* decomposition, we approximate URM:\n",
    "\n",
    "$$ \\widehat{URM} = U_{t} \\Sigma_{t} V^*_{t} $$\n",
    "\n",
    "Where $U_{t}$ is a $m \\times t$ matrix, $\\Sigma_{t}$ is a $t$ vector, and $V^*_{t}$ is a $t \\times n$ matrix. For this approximation, only the $t$ largest singular values are kept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.utils.extmath import randomized_svd\n",
    "\n",
    "# Other SVDs are also available, like from sklearn.decomposition import TruncatedSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "U, Sigma, VT = randomized_svd(URM_train,\n",
    "                              n_components=num_factors,\n",
    "                              random_state=1234)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[ 7.39053964e-04, -2.93912628e-03, -7.78126692e-04, ...,\n",
       "         -7.57091607e-04, -2.20537323e-03, -6.29395109e-04],\n",
       "        [ 6.10583480e-04, -1.40230446e-03, -1.27487887e-04, ...,\n",
       "          3.09211577e-03,  2.39981610e-03,  6.87632605e-04],\n",
       "        [ 5.50076917e-04,  2.22152832e-04, -5.19179339e-04, ...,\n",
       "          6.24366494e-05,  1.50688570e-03,  3.57234614e-04],\n",
       "        ...,\n",
       "        [ 3.42185192e-03,  1.54442175e-03,  5.47862095e-03, ...,\n",
       "         -1.72592588e-03,  2.38431771e-03,  3.72140356e-03],\n",
       "        [ 1.24815540e-03, -4.66883175e-03,  1.04187522e-03, ...,\n",
       "          8.30097296e-04, -2.00077559e-03,  1.99906293e-03],\n",
       "        [ 1.42310674e-03, -7.13603494e-04, -8.82303566e-04, ...,\n",
       "          2.30381742e-03,  1.03186639e-04,  4.95088268e-03]]),\n",
       " (69878, 10))"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U, U.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([4276.64643802, 1784.31557975, 1533.5025626 , 1226.93392479,\n",
       "        1185.4530811 , 1014.45064137,  960.6592356 ,  908.72211896,\n",
       "         842.90214758,  745.34494295]),\n",
       " (10,))"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Sigma, Sigma.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[ 0.00665443,  0.03274073,  0.04184214, ...,  0.        ,\n",
       "          0.        ,  0.        ],\n",
       "        [-0.01478895, -0.09538786, -0.07150558, ..., -0.        ,\n",
       "         -0.        , -0.        ],\n",
       "        [ 0.00297177, -0.01156154, -0.04204534, ..., -0.        ,\n",
       "         -0.        , -0.        ],\n",
       "        ...,\n",
       "        [ 0.00167712, -0.02233305, -0.03248025, ..., -0.        ,\n",
       "         -0.        , -0.        ],\n",
       "        [-0.0062943 , -0.00451829, -0.02969469, ...,  0.        ,\n",
       "          0.        ,  0.        ],\n",
       "        [-0.01288507,  0.01692335, -0.0836164 , ..., -0.        ,\n",
       "         -0.        , -0.        ]]),\n",
       " (10, 10681))"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "VT, VT.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 10681)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "VT.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Computing a prediction\n",
    "\n",
    "So, how do we transform the matrices $U_t$, $\\Sigma_t$, $V^T_t$ into something that we can use for recommendation? \n",
    "\n",
    "Remember, $\\widehat{URM} = U_t \\Sigma_t V^T_t$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Matrix Factorization Approach (PureSVDRecommender)\n",
    "\n",
    "Consider the following matrices $W = U_t \\Sigma_t$, and $H = V^T_t$, then we just have a Matrix Factorization recommender where $\\widehat{URM} = WH$, where $W$ represents the user factors and $H$ are the item factors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import scipy.sparse as sps\n",
    "\n",
    "# Store an intermediate pre-multiplied matrix\n",
    "user_factors = U * sps.diags(Sigma)\n",
    "item_factors = VT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's now predict if Bob would like Toy Story or not..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction is -0.45\n"
     ]
    }
   ],
   "source": [
    "prediction = user_factors[user_index, :].dot(item_factors[:,item_index])\n",
    "\n",
    "print(\"Prediction is {:.2f}\".format(prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "And with this we calculate the score of all items for Bob."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.19917472, -0.26403461, -1.02277717, ...,  0.        ,\n",
       "        0.        ,  0.        ])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_scores = user_factors[user_index, :].dot(item_factors)\n",
    "item_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10681,)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_scores.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So, which are the best 20 items for Bob?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([145, 219, 170, 218, 148, 146, 166, 167, 133, 176,  44, 163, 144,\n",
       "        34, 227, 220, 150, 147, 182, 193])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_items_for_bob = np.flip(np.argsort(item_scores))[:20]\n",
    "best_items_for_bob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Is Toy story inside that list?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_index in best_items_for_bob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Item-Based Approach (PureSVDItemRecommender)\n",
    "\n",
    "Consider the following matrix $P = U_t \\Sigma_t$.\n",
    "\n",
    "As $U$ and $V^T$ are orthogonal (meaning $UU^T = U^TU = I$), then \n",
    "\n",
    "$$ \\widehat{URM} = U_t \\Sigma_t V^T_t $$\n",
    "$$ \\widehat{URM}V = U_t \\Sigma_t V^T_t V $$\n",
    "$$ \\widehat{URM}V = U_t \\Sigma_t $$\n",
    "\n",
    "Re-arranging the equations\n",
    "\n",
    "$$ P = U_t \\Sigma_t = URMV $$\n",
    "\n",
    "With this, if we define $r_u$ as the $u$-th row in the URM and $v^T_i$ as the $i$-th column in $V^T$ then we calculate any $\\hat{r}_{u,i}$ as \n",
    "\n",
    "$$\\hat{r}_{u,i} = r_u V v^T_i$$\n",
    "\n",
    "Which is equivalent to having a similarity matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# BEWARE: This consumes A LOT of memory\n",
    "item_weights = np.dot(VT.T, VT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8.11 s, sys: 1.53 s, total: 9.64 s\n",
      "Wall time: 2.88 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "ITEM_factors = VT.T\n",
    "topK = 100\n",
    "\n",
    "n_items, n_factors = ITEM_factors.shape\n",
    "\n",
    "block_size = 100\n",
    "\n",
    "start_item = 0\n",
    "end_item = 0\n",
    "\n",
    "values = []\n",
    "rows = []\n",
    "cols = []\n",
    "\n",
    "# Compute all similarities for each item using vectorization\n",
    "while start_item < n_items:\n",
    "\n",
    "    end_item = min(n_items, start_item + block_size)\n",
    "\n",
    "    this_block_weight = np.dot(ITEM_factors[start_item:end_item, :], ITEM_factors.T)\n",
    "\n",
    "    for col_index_in_block in range(this_block_weight.shape[0]):\n",
    "\n",
    "        this_column_weights = this_block_weight[col_index_in_block, :]\n",
    "        item_original_index = start_item + col_index_in_block\n",
    "\n",
    "        # Sort indices and select TopK\n",
    "        # Sorting is done in three steps. Faster then plain np.argsort for higher number of items\n",
    "        # - Partition the data to extract the set of relevant items\n",
    "        # - Sort only the relevant items\n",
    "        # - Get the original item index\n",
    "        relevant_items_partition = (-this_column_weights).argpartition(topK-1)[0:topK]\n",
    "        relevant_items_partition_sorting = np.argsort(-this_column_weights[relevant_items_partition])\n",
    "        top_k_idx = relevant_items_partition[relevant_items_partition_sorting]\n",
    "\n",
    "        # Incrementally build sparse matrix, do not add zeros\n",
    "        notZerosMask = this_column_weights[top_k_idx] != 0.0\n",
    "        numNotZeros = np.sum(notZerosMask)\n",
    "\n",
    "        values.extend(this_column_weights[top_k_idx][notZerosMask])\n",
    "        rows.extend(top_k_idx[notZerosMask])\n",
    "        cols.extend(np.ones(numNotZeros) * item_original_index)\n",
    "\n",
    "\n",
    "\n",
    "    start_item += block_size\n",
    "\n",
    "item_weights = sps.csr_matrix((values, (rows, cols)),\n",
    "                          shape=(n_items, n_items),\n",
    "                          dtype=np.float32)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<10681x10681 sparse matrix of type '<class 'numpy.float32'>'\n",
       " \twith 1065800 stored elements in Compressed Sparse Row format>,\n",
       " (10681, 10681))"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_weights, item_weights.shape "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.03041735, 0.27881394, 0.32415883, ..., 0.        , 0.        ,\n",
       "       0.        ])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_scores = URM_train[user_index, :].dot(item_weights).A.flatten()\n",
    "item_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10681,)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_scores.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So, which are the best 20 items for Bob?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 145,  179,   34,  605,  176,  146, 1293,  133,  219,  399,  177,\n",
       "        218,   24,  165,  227,  148, 1008,  170,  166,  140])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_items_for_bob = np.flip(np.argsort(item_scores))[:20]\n",
    "best_items_for_bob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Is Toy story inside that list?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_index in best_items_for_bob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Comparison: BPR, FunkSVD, PureSVD (MF and Item-Based)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from MatrixFactorization.Cython.MatrixFactorization_Cython import MatrixFactorization_BPR_Cython, MatrixFactorization_FunkSVD_Cython\n",
    "from MatrixFactorization.PureSVDRecommender import PureSVDRecommender, PureSVDItemRecommender\n",
    "\n",
    "from Base.Evaluation.Evaluator import EvaluatorHoldout\n",
    "\n",
    "evaluator_test = EvaluatorHoldout(URM_test, cutoff_list=[5, 20])\n",
    "\n",
    "evaluator_validation_early_stopping = EvaluatorHoldout(URM_train, cutoff_list=[5], exclude_seen = False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PureSVDRecommender: URM Detected 23 (0.22 %) cold items.\n",
      "PureSVDRecommender: Computing SVD decomposition...\n",
      "PureSVDRecommender: Computing SVD decomposition... Done!\n",
      "EvaluatorHoldout: Processed 40000 ( 57.31% ) in 30.66 sec. Users per second: 1304\n",
      "EvaluatorHoldout: Processed 69799 ( 100.00% ) in 51.83 sec. Users per second: 1347\n",
      "CPU times: user 1min 41s, sys: 9.66 s, total: 1min 50s\n",
      "Wall time: 57.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "recommender = PureSVDRecommender(URM_train)\n",
    "recommender.fit()\n",
    "\n",
    "result_dict_puresvd, _ = evaluator_test.evaluateRecommender(recommender)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.48879759499897085,\n",
       "  'PRECISION': 0.3588188942535811,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.3659947372694643,\n",
       "  'RECALL': 0.11236172045405454,\n",
       "  'MAP': 0.2902881607664011,\n",
       "  'MRR': 0.5734380148712748,\n",
       "  'NDCG': 0.16199299282251783,\n",
       "  'F1': 0.1711339857000296,\n",
       "  'HIT_RATE': 1.7940944712674967,\n",
       "  'ARHR': 0.9040587974039381,\n",
       "  'NOVELTY': 0.004571292997686143,\n",
       "  'AVERAGE_POPULARITY': 0.3557680500196552,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.978365772661157,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9956703511517455,\n",
       "  'COVERAGE_ITEM': 0.08023593296507818,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.07752083138282932,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.7546581184349866,\n",
       "  'DIVERSITY_GINI': 0.021081071480885498,\n",
       "  'SHANNON_ENTROPY': 8.245711062066341},\n",
       " 20: {'ROC_AUC': 0.5844348823143795,\n",
       "  'PRECISION': 0.2398085932463046,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.34766106830383015,\n",
       "  'RECALL': 0.25745904911809864,\n",
       "  'MAP': 0.20132722886069132,\n",
       "  'MRR': 0.5887065958768947,\n",
       "  'NDCG': 0.25275804790328216,\n",
       "  'F1': 0.24832057076538297,\n",
       "  'HIT_RATE': 4.796171864926432,\n",
       "  'ARHR': 1.1846244857273118,\n",
       "  'NOVELTY': 0.018701446994383992,\n",
       "  'AVERAGE_POPULARITY': 0.31500546613966135,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.945916209567748,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9972951328782717,\n",
       "  'COVERAGE_ITEM': 0.12920138563804887,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.12536279374590395,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.8952460001717278,\n",
       "  'DIVERSITY_GINI': 0.033320276118824405,\n",
       "  'SHANNON_ENTROPY': 8.907628468420295}}"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_dict_puresvd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PureSVDItemRecommender: URM Detected 23 (0.22 %) cold items.\n",
      "PureSVDItemRecommender: Computing SVD decomposition...\n",
      "PureSVDItemRecommender: Computing SVD decomposition... Done!\n",
      "EvaluatorHoldout: Processed 9000 ( 12.89% ) in 32.40 sec. Users per second: 278\n",
      "EvaluatorHoldout: Processed 19000 ( 27.22% ) in 1.08 min. Users per second: 292\n",
      "EvaluatorHoldout: Processed 27000 ( 38.68% ) in 1.59 min. Users per second: 284\n",
      "EvaluatorHoldout: Processed 36000 ( 51.58% ) in 2.15 min. Users per second: 280\n",
      "EvaluatorHoldout: Processed 44000 ( 63.04% ) in 2.65 min. Users per second: 276\n",
      "EvaluatorHoldout: Processed 52000 ( 74.50% ) in 3.19 min. Users per second: 271\n",
      "EvaluatorHoldout: Processed 60000 ( 85.96% ) in 3.69 min. Users per second: 271\n",
      "EvaluatorHoldout: Processed 68000 ( 97.42% ) in 4.20 min. Users per second: 270\n",
      "EvaluatorHoldout: Processed 69799 ( 100.00% ) in 4.33 min. Users per second: 269\n",
      "CPU times: user 6min 44s, sys: 40.8 s, total: 7min 25s\n",
      "Wall time: 6min 28s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "recommender = PureSVDItemRecommender(URM_train)\n",
    "recommender.fit()\n",
    "\n",
    "result_dict_puresvditem, _ = evaluator_test.evaluateRecommender(recommender)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.48895280257119433,\n",
       "  'PRECISION': 0.358268743105283,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.36531803225458637,\n",
       "  'RECALL': 0.1114512892092393,\n",
       "  'MAP': 0.2903381734217877,\n",
       "  'MRR': 0.5739736003858659,\n",
       "  'NDCG': 0.16191103604428606,\n",
       "  'F1': 0.17001409586773145,\n",
       "  'HIT_RATE': 1.7913437155260103,\n",
       "  'ARHR': 0.904660525222383,\n",
       "  'NOVELTY': 0.004571891522989991,\n",
       "  'AVERAGE_POPULARITY': 0.3546708396952197,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.9778826132510097,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9955737206541468,\n",
       "  'COVERAGE_ITEM': 0.08014230877258684,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.07733358299784665,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.7528263545035634,\n",
       "  'DIVERSITY_GINI': 0.020775983819095234,\n",
       "  'SHANNON_ENTROPY': 8.22549615909713},\n",
       " 20: {'ROC_AUC': 0.5843590001947071,\n",
       "  'PRECISION': 0.23961159902002463,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.3470135352495496,\n",
       "  'RECALL': 0.2566079089314121,\n",
       "  'MAP': 0.20103537198833224,\n",
       "  'MRR': 0.5894095745398737,\n",
       "  'NDCG': 0.2526918482436237,\n",
       "  'F1': 0.2478186786088948,\n",
       "  'HIT_RATE': 4.7922319804008655,\n",
       "  'ARHR': 1.1853798210025972,\n",
       "  'NOVELTY': 0.018712583036888067,\n",
       "  'AVERAGE_POPULARITY': 0.31298309206157837,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.945606613560986,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9972796532997104,\n",
       "  'COVERAGE_ITEM': 0.12910776144555752,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.12489467278344724,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.8950313403360142,\n",
       "  'DIVERSITY_GINI': 0.03321143911401798,\n",
       "  'SHANNON_ENTROPY': 8.903700340279745}}"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_dict_puresvditem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MatrixFactorization_BPR_Cython_Recommender: URM Detected 23 (0.22 %) cold items.\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.80 seconds. BPR loss 1.02E-02. Sample per second: 86166\n",
      "MF_BPR: Epoch 1 of 300. Elapsed time 0.52 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.31 seconds. BPR loss 1.01E-02. Sample per second: 52509\n",
      "MF_BPR: Epoch 2 of 300. Elapsed time 1.04 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.85 seconds. BPR loss 1.01E-02. Sample per second: 81237\n",
      "MF_BPR: Epoch 3 of 300. Elapsed time 1.57 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.48 seconds. BPR loss 1.02E-02. Sample per second: 46737\n",
      "MF_BPR: Epoch 4 of 300. Elapsed time 2.20 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.03 seconds. BPR loss 1.02E-02. Sample per second: 66828\n",
      "MF_BPR: Epoch 5 of 300. Elapsed time 2.75 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.54 seconds. BPR loss 1.01E-02. Sample per second: 127321\n",
      "MF_BPR: Epoch 6 of 300. Elapsed time 3.26 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.03 seconds. BPR loss 1.01E-02. Sample per second: 67175\n",
      "MF_BPR: Epoch 7 of 300. Elapsed time 3.75 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.51 seconds. BPR loss 1.03E-02. Sample per second: 135011\n",
      "MF_BPR: Epoch 8 of 300. Elapsed time 4.23 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.15 seconds. BPR loss 1.02E-02. Sample per second: 59978\n",
      "MF_BPR: Epoch 9 of 300. Elapsed time 4.87 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.71 seconds. BPR loss 1.02E-02. Sample per second: 96768\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.12% ) in 30.43 sec. Users per second: 1610\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 44.43 sec. Users per second: 1573\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257222, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052864, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253074, NOVELTY: 0.0074978, AVERAGE_POPULARITY: 0.0274489, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062744, SHANNON_ENTROPY: 12.4778604, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 10 of 300. Elapsed time 50.00 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.98 seconds. BPR loss 1.02E-02. Sample per second: 70167\n",
      "MF_BPR: Epoch 11 of 300. Elapsed time 50.71 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.68 seconds. BPR loss 1.01E-02. Sample per second: 41050\n",
      "MF_BPR: Epoch 12 of 300. Elapsed time 51.40 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.29 seconds. BPR loss 1.02E-02. Sample per second: 53346\n",
      "MF_BPR: Epoch 13 of 300. Elapsed time 52.02 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.94 seconds. BPR loss 1.02E-02. Sample per second: 73224\n",
      "MF_BPR: Epoch 14 of 300. Elapsed time 52.66 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.55 seconds. BPR loss 1.02E-02. Sample per second: 44403\n",
      "MF_BPR: Epoch 15 of 300. Elapsed time 53.28 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.13 seconds. BPR loss 1.03E-02. Sample per second: 61251\n",
      "MF_BPR: Epoch 16 of 300. Elapsed time 53.85 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.67 seconds. BPR loss 1.01E-02. Sample per second: 103421\n",
      "MF_BPR: Epoch 17 of 300. Elapsed time 54.39 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.34 seconds. BPR loss 1.03E-02. Sample per second: 51618\n",
      "MF_BPR: Epoch 18 of 300. Elapsed time 55.06 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.90 seconds. BPR loss 1.01E-02. Sample per second: 76620\n",
      "MF_BPR: Epoch 19 of 300. Elapsed time 55.62 sec\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.48 seconds. BPR loss 1.02E-02. Sample per second: 46639\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 44000 ( 62.97% ) in 30.52 sec. Users per second: 1442\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 45.91 sec. Users per second: 1522\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257222, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052864, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253074, NOVELTY: 0.0074978, AVERAGE_POPULARITY: 0.0274447, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062760, SHANNON_ENTROPY: 12.4778648, \n",
      "\n",
      "MF_BPR: Epoch 20 of 300. Elapsed time 1.70 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.06 seconds. BPR loss 1.02E-02. Sample per second: 64837\n",
      "MF_BPR: Epoch 21 of 300. Elapsed time 1.71 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.68 seconds. BPR loss 1.02E-02. Sample per second: 101540\n",
      "MF_BPR: Epoch 22 of 300. Elapsed time 1.72 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.50 seconds. BPR loss 1.01E-02. Sample per second: 46136\n",
      "MF_BPR: Epoch 23 of 300. Elapsed time 1.74 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.16 seconds. BPR loss 1.02E-02. Sample per second: 59343\n",
      "MF_BPR: Epoch 24 of 300. Elapsed time 1.75 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.85 seconds. BPR loss 1.02E-02. Sample per second: 80736\n",
      "MF_BPR: Epoch 25 of 300. Elapsed time 1.76 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.64 seconds. BPR loss 1.02E-02. Sample per second: 42127\n",
      "MF_BPR: Epoch 26 of 300. Elapsed time 1.77 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.51 seconds. BPR loss 1.02E-02. Sample per second: 45546\n",
      "MF_BPR: Epoch 27 of 300. Elapsed time 1.79 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.36 seconds. BPR loss 1.01E-02. Sample per second: 50561\n",
      "MF_BPR: Epoch 28 of 300. Elapsed time 1.80 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.29 seconds. BPR loss 1.02E-02. Sample per second: 53364\n",
      "MF_BPR: Epoch 29 of 300. Elapsed time 1.82 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.09 seconds. BPR loss 1.01E-02. Sample per second: 63223\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 39000 ( 55.81% ) in 30.03 sec. Users per second: 1299\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 47.14 sec. Users per second: 1482\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257246, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052868, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253082, NOVELTY: 0.0074978, AVERAGE_POPULARITY: 0.0274453, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062764, SHANNON_ENTROPY: 12.4778646, \n",
      "\n",
      "MF_BPR: New best model found! Updating.\n",
      "MF_BPR: Epoch 30 of 300. Elapsed time 2.62 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.78 seconds. BPR loss 1.00E-02. Sample per second: 87975\n",
      "MF_BPR: Epoch 31 of 300. Elapsed time 2.63 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.24 seconds. BPR loss 1.02E-02. Sample per second: 55716\n",
      "MF_BPR: Epoch 32 of 300. Elapsed time 2.63 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.69 seconds. BPR loss 1.01E-02. Sample per second: 99295\n",
      "MF_BPR: Epoch 33 of 300. Elapsed time 2.64 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.15 seconds. BPR loss 1.01E-02. Sample per second: 60203\n",
      "MF_BPR: Epoch 34 of 300. Elapsed time 2.65 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.62 seconds. BPR loss 1.02E-02. Sample per second: 110485\n",
      "MF_BPR: Epoch 35 of 300. Elapsed time 2.66 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.07 seconds. BPR loss 1.02E-02. Sample per second: 64308\n",
      "MF_BPR: Epoch 36 of 300. Elapsed time 2.66 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.53 seconds. BPR loss 1.02E-02. Sample per second: 131029\n",
      "MF_BPR: Epoch 37 of 300. Elapsed time 2.67 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.98 seconds. BPR loss 1.02E-02. Sample per second: 70285\n",
      "MF_BPR: Epoch 38 of 300. Elapsed time 2.68 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.44 seconds. BPR loss 1.02E-02. Sample per second: 47904\n",
      "MF_BPR: Epoch 39 of 300. Elapsed time 2.69 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.89 seconds. BPR loss 1.03E-02. Sample per second: 77221\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 60000 ( 85.86% ) in 30.08 sec. Users per second: 1995\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 35.08 sec. Users per second: 1992\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257222, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052864, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253074, NOVELTY: 0.0074978, AVERAGE_POPULARITY: 0.0274453, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062772, SHANNON_ENTROPY: 12.4778670, \n",
      "\n",
      "MF_BPR: Epoch 40 of 300. Elapsed time 3.28 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.47 seconds. BPR loss 1.01E-02. Sample per second: 46842\n",
      "MF_BPR: Epoch 41 of 300. Elapsed time 3.29 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.94 seconds. BPR loss 1.02E-02. Sample per second: 73512\n",
      "MF_BPR: Epoch 42 of 300. Elapsed time 3.29 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.39 seconds. BPR loss 1.02E-02. Sample per second: 49651\n",
      "MF_BPR: Epoch 43 of 300. Elapsed time 3.30 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.84 seconds. BPR loss 1.01E-02. Sample per second: 82294\n",
      "MF_BPR: Epoch 44 of 300. Elapsed time 3.31 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.28 seconds. BPR loss 1.01E-02. Sample per second: 53800\n",
      "MF_BPR: Epoch 45 of 300. Elapsed time 3.32 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.73 seconds. BPR loss 1.01E-02. Sample per second: 94114\n",
      "MF_BPR: Epoch 46 of 300. Elapsed time 3.32 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.18 seconds. BPR loss 1.03E-02. Sample per second: 58519\n",
      "MF_BPR: Epoch 47 of 300. Elapsed time 3.33 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.62 seconds. BPR loss 1.01E-02. Sample per second: 111486\n",
      "MF_BPR: Epoch 48 of 300. Elapsed time 3.34 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.08 seconds. BPR loss 1.01E-02. Sample per second: 63960\n",
      "MF_BPR: Epoch 49 of 300. Elapsed time 3.35 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.52 seconds. BPR loss 1.01E-02. Sample per second: 131814\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 62000 ( 88.73% ) in 30.33 sec. Users per second: 2044\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 34.12 sec. Users per second: 2048\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257246, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052868, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253082, NOVELTY: 0.0074978, AVERAGE_POPULARITY: 0.0274443, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062721, SHANNON_ENTROPY: 12.4778513, \n",
      "\n",
      "MF_BPR: Epoch 50 of 300. Elapsed time 3.92 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.18 seconds. BPR loss 1.02E-02. Sample per second: 58550\n",
      "MF_BPR: Epoch 51 of 300. Elapsed time 3.93 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.65 seconds. BPR loss 1.01E-02. Sample per second: 106178\n",
      "MF_BPR: Epoch 52 of 300. Elapsed time 3.94 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.11 seconds. BPR loss 1.02E-02. Sample per second: 62401\n",
      "MF_BPR: Epoch 53 of 300. Elapsed time 3.95 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.55 seconds. BPR loss 1.02E-02. Sample per second: 125261\n",
      "MF_BPR: Epoch 54 of 300. Elapsed time 3.95 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.99 seconds. BPR loss 1.01E-02. Sample per second: 69417\n",
      "MF_BPR: Epoch 55 of 300. Elapsed time 3.96 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.44 seconds. BPR loss 1.02E-02. Sample per second: 47817\n",
      "MF_BPR: Epoch 56 of 300. Elapsed time 3.97 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.88 seconds. BPR loss 1.01E-02. Sample per second: 78101\n",
      "MF_BPR: Epoch 57 of 300. Elapsed time 3.98 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.32 seconds. BPR loss 1.01E-02. Sample per second: 52099\n",
      "MF_BPR: Epoch 58 of 300. Elapsed time 3.98 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.77 seconds. BPR loss 1.02E-02. Sample per second: 89928\n",
      "MF_BPR: Epoch 59 of 300. Elapsed time 3.99 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.22 seconds. BPR loss 1.02E-02. Sample per second: 56532\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.69% ) in 30.12 sec. Users per second: 1593\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 42.15 sec. Users per second: 1658\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257222, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052864, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253074, NOVELTY: 0.0074978, AVERAGE_POPULARITY: 0.0274443, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062725, SHANNON_ENTROPY: 12.4778543, \n",
      "\n",
      "MF_BPR: Epoch 60 of 300. Elapsed time 4.70 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.93 seconds. BPR loss 1.03E-02. Sample per second: 74032\n",
      "MF_BPR: Epoch 61 of 300. Elapsed time 4.71 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.43 seconds. BPR loss 1.01E-02. Sample per second: 48296\n",
      "MF_BPR: Epoch 62 of 300. Elapsed time 4.72 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.91 seconds. BPR loss 1.02E-02. Sample per second: 75823\n",
      "MF_BPR: Epoch 63 of 300. Elapsed time 4.73 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.37 seconds. BPR loss 1.01E-02. Sample per second: 50542\n",
      "MF_BPR: Epoch 64 of 300. Elapsed time 4.73 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.82 seconds. BPR loss 1.01E-02. Sample per second: 83866\n",
      "MF_BPR: Epoch 65 of 300. Elapsed time 4.74 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.29 seconds. BPR loss 1.02E-02. Sample per second: 53683\n",
      "MF_BPR: Epoch 66 of 300. Elapsed time 4.75 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.74 seconds. BPR loss 1.02E-02. Sample per second: 93768\n",
      "MF_BPR: Epoch 67 of 300. Elapsed time 4.76 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.20 seconds. BPR loss 1.02E-02. Sample per second: 57369\n",
      "MF_BPR: Epoch 68 of 300. Elapsed time 4.77 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.66 seconds. BPR loss 1.02E-02. Sample per second: 105148\n",
      "MF_BPR: Epoch 69 of 300. Elapsed time 4.77 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.16 seconds. BPR loss 1.02E-02. Sample per second: 59404\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 57000 ( 81.57% ) in 30.41 sec. Users per second: 1874\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 36.94 sec. Users per second: 1892\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257222, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052864, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253074, NOVELTY: 0.0074978, AVERAGE_POPULARITY: 0.0274443, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062739, SHANNON_ENTROPY: 12.4778589, \n",
      "\n",
      "MF_BPR: Epoch 70 of 300. Elapsed time 5.40 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.65 seconds. BPR loss 1.03E-02. Sample per second: 105923\n",
      "MF_BPR: Epoch 71 of 300. Elapsed time 5.41 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.15 seconds. BPR loss 1.01E-02. Sample per second: 59745\n",
      "MF_BPR: Epoch 72 of 300. Elapsed time 5.41 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.63 seconds. BPR loss 1.02E-02. Sample per second: 108833\n",
      "MF_BPR: Epoch 73 of 300. Elapsed time 5.42 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.10 seconds. BPR loss 1.02E-02. Sample per second: 62992\n",
      "MF_BPR: Epoch 74 of 300. Elapsed time 5.43 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.56 seconds. BPR loss 1.01E-02. Sample per second: 123691\n",
      "MF_BPR: Epoch 75 of 300. Elapsed time 5.44 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.02 seconds. BPR loss 1.01E-02. Sample per second: 67863\n",
      "MF_BPR: Epoch 76 of 300. Elapsed time 5.45 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.47 seconds. BPR loss 1.02E-02. Sample per second: 147418\n",
      "MF_BPR: Epoch 77 of 300. Elapsed time 5.45 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.92 seconds. BPR loss 1.02E-02. Sample per second: 74964\n",
      "MF_BPR: Epoch 78 of 300. Elapsed time 5.46 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 1.40 seconds. BPR loss 1.01E-02. Sample per second: 49377\n",
      "MF_BPR: Epoch 79 of 300. Elapsed time 5.47 min\n",
      "MF_BPR: Processed 69000 ( 98.57% ) in 0.85 seconds. BPR loss 1.01E-02. Sample per second: 81083\n",
      "MF_BPR: Validation begins...\n",
      "EvaluatorHoldout: Processed 58000 ( 83.00% ) in 30.12 sec. Users per second: 1926\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 36.00 sec. Users per second: 1941\n",
      "MF_BPR: CUTOFF: 5 - ROC_AUC: 0.0257222, PRECISION: 0.0109620, PRECISION_RECALL_MIN_DEN: 0.0109620, RECALL: 0.0004912, MAP: 0.0052864, MRR: 0.0242828, NDCG: 0.0010555, F1: 0.0009402, HIT_RATE: 0.0548098, ARHR: 0.0253074, NOVELTY: 0.0074979, AVERAGE_POPULARITY: 0.0274438, DIVERSITY_MEAN_INTER_LIST: 0.9987407, DIVERSITY_HERFINDAHL: 0.9997453, COVERAGE_ITEM: 0.9397996, COVERAGE_ITEM_CORRECT: 0.1332272, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.0510175, DIVERSITY_GINI: 0.4062785, SHANNON_ENTROPY: 12.4778748, \n",
      "\n",
      "MF_BPR: Convergence reached! Terminating at epoch 80. Best value for 'MAP' at epoch 30 is 0.0053. Elapsed time 6.08 min\n",
      "MF_BPR: Epoch 80 of 300. Elapsed time 6.08 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 42000 ( 60.17% ) in 30.69 sec. Users per second: 1369\n",
      "EvaluatorHoldout: Processed 69799 ( 100.00% ) in 50.54 sec. Users per second: 1381\n",
      "CPU times: user 11min 37s, sys: 1min 12s, total: 12min 49s\n",
      "Wall time: 6min 56s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "recommender = MatrixFactorization_BPR_Cython(URM_train)\n",
    "recommender.fit(num_factors = 50, \n",
    "                validation_every_n = 10, \n",
    "                stop_on_validation = True, \n",
    "                evaluator_object = evaluator_validation_early_stopping,\n",
    "                lower_validations_allowed = 5, \n",
    "                validation_metric = \"MAP\")\n",
    "\n",
    "result_dict_bpr, _ = evaluator_test.evaluateRecommender(recommender)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.0066679560834204885,\n",
       "  'PRECISION': 0.002727832776973851,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.0027526659885289354,\n",
       "  'RECALL': 0.0005234563207056631,\n",
       "  'MAP': 0.0012606834545544291,\n",
       "  'MRR': 0.006124490799772649,\n",
       "  'NDCG': 0.0006706927944033529,\n",
       "  'F1': 0.000878360100277859,\n",
       "  'HIT_RATE': 0.013639163884869411,\n",
       "  'ARHR': 0.0061877677330620445,\n",
       "  'NOVELTY': 0.007523037993188253,\n",
       "  'AVERAGE_POPULARITY': 0.025128165704133305,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.9987433318396813,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9997458045981922,\n",
       "  'COVERAGE_ITEM': 0.9403613893830166,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.05692350903473457,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.013409084404247403,\n",
       "  'DIVERSITY_GINI': 0.40698870372049323,\n",
       "  'SHANNON_ENTROPY': 12.480280868232452},\n",
       " 20: {'ROC_AUC': 0.02504328745491179,\n",
       "  'PRECISION': 0.002770096992793701,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.003412251399730683,\n",
       "  'RECALL': 0.0019263672614757638,\n",
       "  'MAP': 0.0006398198636900119,\n",
       "  'MRR': 0.009418507195707016,\n",
       "  'NDCG': 0.0015627496210535725,\n",
       "  'F1': 0.002272443212222553,\n",
       "  'HIT_RATE': 0.055401939855871864,\n",
       "  'ARHR': 0.009844734866369801,\n",
       "  'NOVELTY': 0.03011533065566006,\n",
       "  'AVERAGE_POPULARITY': 0.02499363270897789,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.9963767145161505,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9998181219786804,\n",
       "  'COVERAGE_ITEM': 0.995318790375433,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.15288830633835784,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.05043075073699877,\n",
       "  'DIVERSITY_GINI': 0.5134604053468222,\n",
       "  'SHANNON_ENTROPY': 12.806972967233468}}"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_dict_bpr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 23 (0.22 %) cold items.\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.92 seconds. MSE loss 1.95E+00. Sample per second: 222766\n",
      "FUNK_SVD: Epoch 1 of 300. Elapsed time 34.93 sec\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.86 seconds. MSE loss 1.14E+00. Sample per second: 223136\n",
      "FUNK_SVD: Epoch 2 of 300. Elapsed time 1.16 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.71 seconds. MSE loss 1.13E+00. Sample per second: 224065\n",
      "FUNK_SVD: Epoch 3 of 300. Elapsed time 1.75 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.18 seconds. MSE loss 1.13E+00. Sample per second: 221168\n",
      "FUNK_SVD: Epoch 4 of 300. Elapsed time 2.34 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.31 seconds. MSE loss 1.13E+00. Sample per second: 226616\n",
      "FUNK_SVD: Epoch 5 of 300. Elapsed time 2.92 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.55 seconds. MSE loss 1.12E+00. Sample per second: 225073\n",
      "FUNK_SVD: Epoch 6 of 300. Elapsed time 3.51 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.52 seconds. MSE loss 1.12E+00. Sample per second: 225273\n",
      "FUNK_SVD: Epoch 7 of 300. Elapsed time 4.09 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.37 seconds. MSE loss 1.12E+00. Sample per second: 226219\n",
      "FUNK_SVD: Epoch 8 of 300. Elapsed time 4.67 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.39 seconds. MSE loss 1.11E+00. Sample per second: 226130\n",
      "FUNK_SVD: Epoch 9 of 300. Elapsed time 5.26 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.12 seconds. MSE loss 1.11E+00. Sample per second: 221512\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.71% ) in 30.24 sec. Users per second: 1818\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 38.19 sec. Users per second: 1830\n",
      "FUNK_SVD: CUTOFF: 5 - ROC_AUC: 0.4073969, PRECISION: 0.2491943, PRECISION_RECALL_MIN_DEN: 0.2491943, RECALL: 0.0190450, MAP: 0.1829174, MRR: 0.4550657, NDCG: 0.0564623, F1: 0.0353856, HIT_RATE: 1.2459716, ARHR: 0.6479994, NOVELTY: 0.0048558, AVERAGE_POPULARITY: 0.6257405, DIVERSITY_MEAN_INTER_LIST: 0.7000088, DIVERSITY_HERFINDAHL: 0.9399997, COVERAGE_ITEM: 0.7533003, COVERAGE_ITEM_CORRECT: 0.0547702, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.6439366, DIVERSITY_GINI: 0.0801234, SHANNON_ENTROPY: 6.7503694, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 10 of 300. Elapsed time 6.49 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.14 seconds. MSE loss 1.11E+00. Sample per second: 227704\n",
      "FUNK_SVD: Epoch 11 of 300. Elapsed time 7.07 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.45 seconds. MSE loss 1.10E+00. Sample per second: 225747\n",
      "FUNK_SVD: Epoch 12 of 300. Elapsed time 7.66 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.40 seconds. MSE loss 1.10E+00. Sample per second: 226025\n",
      "FUNK_SVD: Epoch 13 of 300. Elapsed time 8.24 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.66 seconds. MSE loss 1.10E+00. Sample per second: 218285\n",
      "FUNK_SVD: Epoch 14 of 300. Elapsed time 8.84 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.99 seconds. MSE loss 1.10E+00. Sample per second: 205249\n",
      "FUNK_SVD: Epoch 15 of 300. Elapsed time 9.48 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 39.01 seconds. MSE loss 1.10E+00. Sample per second: 205120\n",
      "FUNK_SVD: Epoch 16 of 300. Elapsed time 10.12 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 41.45 seconds. MSE loss 1.10E+00. Sample per second: 193051\n",
      "FUNK_SVD: Epoch 17 of 300. Elapsed time 10.81 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 39.19 seconds. MSE loss 1.09E+00. Sample per second: 204180\n",
      "FUNK_SVD: Epoch 18 of 300. Elapsed time 11.45 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.33 seconds. MSE loss 1.09E+00. Sample per second: 226472\n",
      "FUNK_SVD: Epoch 19 of 300. Elapsed time 12.04 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.62 seconds. MSE loss 1.09E+00. Sample per second: 218506\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 48000 ( 68.69% ) in 30.78 sec. Users per second: 1559\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 45.65 sec. Users per second: 1531\n",
      "FUNK_SVD: CUTOFF: 5 - ROC_AUC: 0.3891115, PRECISION: 0.3316695, PRECISION_RECALL_MIN_DEN: 0.3316695, RECALL: 0.0253962, MAP: 0.2421203, MRR: 0.4885286, NDCG: 0.0713544, F1: 0.0471798, HIT_RATE: 1.6583474, ARHR: 0.7736371, NOVELTY: 0.0039659, AVERAGE_POPULARITY: 0.8331588, DIVERSITY_MEAN_INTER_LIST: 0.4322206, DIVERSITY_HERFINDAHL: 0.8864429, COVERAGE_ITEM: 0.0779890, COVERAGE_ITEM_CORRECT: 0.0072091, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.7172787, DIVERSITY_GINI: 0.0011135, SHANNON_ENTROPY: 3.5707357, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 20 of 300. Elapsed time 13.41 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.69 seconds. MSE loss 1.09E+00. Sample per second: 218083\n",
      "FUNK_SVD: Epoch 21 of 300. Elapsed time 14.01 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.44 seconds. MSE loss 1.08E+00. Sample per second: 208168\n",
      "FUNK_SVD: Epoch 22 of 300. Elapsed time 14.64 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.48 seconds. MSE loss 1.08E+00. Sample per second: 219364\n",
      "FUNK_SVD: Epoch 23 of 300. Elapsed time 15.24 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.04 seconds. MSE loss 1.08E+00. Sample per second: 216020\n",
      "FUNK_SVD: Epoch 24 of 300. Elapsed time 15.85 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.90 seconds. MSE loss 1.08E+00. Sample per second: 216859\n",
      "FUNK_SVD: Epoch 25 of 300. Elapsed time 16.47 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.85 seconds. MSE loss 1.08E+00. Sample per second: 223218\n",
      "FUNK_SVD: Epoch 26 of 300. Elapsed time 17.05 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.66 seconds. MSE loss 1.08E+00. Sample per second: 218260\n",
      "FUNK_SVD: Epoch 27 of 300. Elapsed time 17.64 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 34.90 seconds. MSE loss 1.08E+00. Sample per second: 229277\n",
      "FUNK_SVD: Epoch 28 of 300. Elapsed time 18.22 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.78 seconds. MSE loss 1.08E+00. Sample per second: 223664\n",
      "FUNK_SVD: Epoch 29 of 300. Elapsed time 18.80 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.30 seconds. MSE loss 1.07E+00. Sample per second: 208912\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 49000 ( 70.12% ) in 30.07 sec. Users per second: 1629\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 42.19 sec. Users per second: 1656\n",
      "FUNK_SVD: CUTOFF: 5 - ROC_AUC: 0.3855148, PRECISION: 0.3290649, PRECISION_RECALL_MIN_DEN: 0.3290649, RECALL: 0.0250226, MAP: 0.2397205, MRR: 0.4866963, NDCG: 0.0712643, F1: 0.0465087, HIT_RATE: 1.6453247, ARHR: 0.7679782, NOVELTY: 0.0039614, AVERAGE_POPULARITY: 0.8267808, DIVERSITY_MEAN_INTER_LIST: 0.3760482, DIVERSITY_HERFINDAHL: 0.8752086, COVERAGE_ITEM: 0.0067409, COVERAGE_ITEM_CORRECT: 0.0042131, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.7154183, DIVERSITY_GINI: 0.0008055, SHANNON_ENTROPY: 3.3319253, \n",
      "\n",
      "FUNK_SVD: Epoch 30 of 300. Elapsed time 20.13 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.02 seconds. MSE loss 1.07E+00. Sample per second: 210468\n",
      "FUNK_SVD: Epoch 31 of 300. Elapsed time 20.75 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 40.85 seconds. MSE loss 1.07E+00. Sample per second: 195882\n",
      "FUNK_SVD: Epoch 32 of 300. Elapsed time 21.43 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 39.49 seconds. MSE loss 1.07E+00. Sample per second: 202632\n",
      "FUNK_SVD: Epoch 33 of 300. Elapsed time 22.08 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 40.45 seconds. MSE loss 1.07E+00. Sample per second: 197817\n",
      "FUNK_SVD: Epoch 34 of 300. Elapsed time 22.74 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.00 seconds. MSE loss 1.07E+00. Sample per second: 216258\n",
      "FUNK_SVD: Epoch 35 of 300. Elapsed time 23.35 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.08 seconds. MSE loss 1.07E+00. Sample per second: 228083\n",
      "FUNK_SVD: Epoch 36 of 300. Elapsed time 23.93 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 34.39 seconds. MSE loss 1.07E+00. Sample per second: 232683\n",
      "FUNK_SVD: Epoch 37 of 300. Elapsed time 24.51 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.45 seconds. MSE loss 1.06E+00. Sample per second: 225717\n",
      "FUNK_SVD: Epoch 38 of 300. Elapsed time 25.09 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.98 seconds. MSE loss 1.06E+00. Sample per second: 222415\n",
      "FUNK_SVD: Epoch 39 of 300. Elapsed time 25.68 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 34.83 seconds. MSE loss 1.06E+00. Sample per second: 229748\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 57000 ( 81.57% ) in 30.45 sec. Users per second: 1872\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 37.38 sec. Users per second: 1869\n",
      "FUNK_SVD: CUTOFF: 5 - ROC_AUC: 0.3832978, PRECISION: 0.3229972, PRECISION_RECALL_MIN_DEN: 0.3229972, RECALL: 0.0243043, MAP: 0.2347366, MRR: 0.4836985, NDCG: 0.0702302, F1: 0.0452070, HIT_RATE: 1.6149861, ARHR: 0.7570428, NOVELTY: 0.0039743, AVERAGE_POPULARITY: 0.8115235, DIVERSITY_MEAN_INTER_LIST: 0.3462916, DIVERSITY_HERFINDAHL: 0.8692573, COVERAGE_ITEM: 0.0036513, COVERAGE_ITEM_CORRECT: 0.0033705, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.7114256, DIVERSITY_GINI: 0.0007632, SHANNON_ENTROPY: 3.2455180, \n",
      "\n",
      "FUNK_SVD: Epoch 40 of 300. Elapsed time 26.87 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 34.70 seconds. MSE loss 1.06E+00. Sample per second: 230637\n",
      "FUNK_SVD: Epoch 41 of 300. Elapsed time 27.45 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.23 seconds. MSE loss 1.06E+00. Sample per second: 227106\n",
      "FUNK_SVD: Epoch 42 of 300. Elapsed time 28.02 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.38 seconds. MSE loss 1.06E+00. Sample per second: 208508\n",
      "FUNK_SVD: Epoch 43 of 300. Elapsed time 28.66 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.16 seconds. MSE loss 1.06E+00. Sample per second: 227604\n",
      "FUNK_SVD: Epoch 44 of 300. Elapsed time 29.24 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.32 seconds. MSE loss 1.06E+00. Sample per second: 214401\n",
      "FUNK_SVD: Epoch 45 of 300. Elapsed time 29.86 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.95 seconds. MSE loss 1.06E+00. Sample per second: 222591\n",
      "FUNK_SVD: Epoch 46 of 300. Elapsed time 30.45 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.33 seconds. MSE loss 1.06E+00. Sample per second: 220250\n",
      "FUNK_SVD: Epoch 47 of 300. Elapsed time 31.04 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.54 seconds. MSE loss 1.06E+00. Sample per second: 225174\n",
      "FUNK_SVD: Epoch 48 of 300. Elapsed time 31.63 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.02 seconds. MSE loss 1.05E+00. Sample per second: 228509\n",
      "FUNK_SVD: Epoch 49 of 300. Elapsed time 32.20 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 34.63 seconds. MSE loss 1.05E+00. Sample per second: 231066\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 52000 ( 74.42% ) in 30.44 sec. Users per second: 1708\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 41.40 sec. Users per second: 1688\n",
      "FUNK_SVD: CUTOFF: 5 - ROC_AUC: 0.3829281, PRECISION: 0.3164830, PRECISION_RECALL_MIN_DEN: 0.3164830, RECALL: 0.0235913, MAP: 0.2294552, MRR: 0.4809270, NDCG: 0.0690833, F1: 0.0439095, HIT_RATE: 1.5824151, ARHR: 0.7457972, NOVELTY: 0.0039879, AVERAGE_POPULARITY: 0.7960138, DIVERSITY_MEAN_INTER_LIST: 0.3209413, DIVERSITY_HERFINDAHL: 0.8641873, COVERAGE_ITEM: 0.0035577, COVERAGE_ITEM_CORRECT: 0.0030896, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.7079624, DIVERSITY_GINI: 0.0007435, SHANNON_ENTROPY: 3.1781597, \n",
      "\n",
      "FUNK_SVD: Epoch 50 of 300. Elapsed time 33.47 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.23 seconds. MSE loss 1.05E+00. Sample per second: 220881\n",
      "FUNK_SVD: Epoch 51 of 300. Elapsed time 34.07 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 39.27 seconds. MSE loss 1.05E+00. Sample per second: 203771\n",
      "FUNK_SVD: Epoch 52 of 300. Elapsed time 34.72 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.51 seconds. MSE loss 1.05E+00. Sample per second: 213302\n",
      "FUNK_SVD: Epoch 53 of 300. Elapsed time 35.34 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.96 seconds. MSE loss 1.05E+00. Sample per second: 222540\n",
      "FUNK_SVD: Epoch 54 of 300. Elapsed time 35.93 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.05 seconds. MSE loss 1.05E+00. Sample per second: 221965\n",
      "FUNK_SVD: Epoch 55 of 300. Elapsed time 36.52 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.55 seconds. MSE loss 1.05E+00. Sample per second: 225084\n",
      "FUNK_SVD: Epoch 56 of 300. Elapsed time 37.11 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.54 seconds. MSE loss 1.05E+00. Sample per second: 218993\n",
      "FUNK_SVD: Epoch 57 of 300. Elapsed time 37.71 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.70 seconds. MSE loss 1.05E+00. Sample per second: 206785\n",
      "FUNK_SVD: Epoch 58 of 300. Elapsed time 38.35 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.04 seconds. MSE loss 1.05E+00. Sample per second: 222045\n",
      "FUNK_SVD: Epoch 59 of 300. Elapsed time 38.93 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.62 seconds. MSE loss 1.05E+00. Sample per second: 212685\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 55000 ( 78.71% ) in 30.49 sec. Users per second: 1804\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 38.28 sec. Users per second: 1825\n",
      "FUNK_SVD: CUTOFF: 5 - ROC_AUC: 0.3807159, PRECISION: 0.3111137, PRECISION_RECALL_MIN_DEN: 0.3111137, RECALL: 0.0229548, MAP: 0.2249487, MRR: 0.4779232, NDCG: 0.0679267, F1: 0.0427550, HIT_RATE: 1.5555683, ARHR: 0.7356662, NOVELTY: 0.0040002, AVERAGE_POPULARITY: 0.7820967, DIVERSITY_MEAN_INTER_LIST: 0.2993676, DIVERSITY_HERFINDAHL: 0.8598727, COVERAGE_ITEM: 0.0035577, COVERAGE_ITEM_CORRECT: 0.0030896, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.7040556, DIVERSITY_GINI: 0.0007231, SHANNON_ENTROPY: 3.1263835, \n",
      "\n",
      "FUNK_SVD: Epoch 60 of 300. Elapsed time 40.20 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 39.34 seconds. MSE loss 1.05E+00. Sample per second: 203383\n",
      "FUNK_SVD: Epoch 61 of 300. Elapsed time 40.84 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.43 seconds. MSE loss 1.05E+00. Sample per second: 208223\n",
      "FUNK_SVD: Epoch 62 of 300. Elapsed time 41.47 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 38.05 seconds. MSE loss 1.04E+00. Sample per second: 210320\n",
      "FUNK_SVD: Epoch 63 of 300. Elapsed time 42.10 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.34 seconds. MSE loss 1.04E+00. Sample per second: 220204\n",
      "FUNK_SVD: Epoch 64 of 300. Elapsed time 42.71 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.04 seconds. MSE loss 1.04E+00. Sample per second: 216019\n",
      "FUNK_SVD: Epoch 65 of 300. Elapsed time 43.32 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.06 seconds. MSE loss 1.04E+00. Sample per second: 215945\n",
      "FUNK_SVD: Epoch 66 of 300. Elapsed time 43.93 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 37.22 seconds. MSE loss 1.04E+00. Sample per second: 214990\n",
      "FUNK_SVD: Epoch 67 of 300. Elapsed time 44.55 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.46 seconds. MSE loss 1.04E+00. Sample per second: 219473\n",
      "FUNK_SVD: Epoch 68 of 300. Elapsed time 45.16 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 35.43 seconds. MSE loss 1.04E+00. Sample per second: 225870\n",
      "FUNK_SVD: Epoch 69 of 300. Elapsed time 45.74 min\n",
      "FUNK_SVD: Processed 8002000 ( 99.99% ) in 36.91 seconds. MSE loss 1.04E+00. Sample per second: 216807\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 54000 ( 77.28% ) in 30.41 sec. Users per second: 1776\n",
      "EvaluatorHoldout: Processed 69878 ( 100.00% ) in 38.87 sec. Users per second: 1798\n",
      "FUNK_SVD: CUTOFF: 5 - ROC_AUC: 0.3788520, PRECISION: 0.3066430, PRECISION_RECALL_MIN_DEN: 0.3066430, RECALL: 0.0224444, MAP: 0.2213691, MRR: 0.4750439, NDCG: 0.0670005, F1: 0.0418272, HIT_RATE: 1.5332150, ARHR: 0.7272315, NOVELTY: 0.0040104, AVERAGE_POPULARITY: 0.7707108, DIVERSITY_MEAN_INTER_LIST: 0.2808936, DIVERSITY_HERFINDAHL: 0.8561779, COVERAGE_ITEM: 0.0036513, COVERAGE_ITEM_CORRECT: 0.0030896, COVERAGE_USER: 1.0000000, COVERAGE_USER_CORRECT: 0.6994905, DIVERSITY_GINI: 0.0007058, SHANNON_ENTROPY: 3.0899581, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 70. Best value for 'MAP' at epoch 20 is 0.2421. Elapsed time 47.00 min\n",
      "FUNK_SVD: Epoch 70 of 300. Elapsed time 47.00 min\n",
      "EvaluatorHoldout: Processed 37000 ( 53.01% ) in 30.45 sec. Users per second: 1215\n",
      "EvaluatorHoldout: Processed 69799 ( 100.00% ) in 55.29 sec. Users per second: 1262\n",
      "CPU times: user 52min 2s, sys: 1min 22s, total: 53min 25s\n",
      "Wall time: 47min 55s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "recommender = MatrixFactorization_FunkSVD_Cython(URM_train)\n",
    "recommender.fit(num_factors = 50, \n",
    "                validation_every_n = 10, \n",
    "                stop_on_validation = True, \n",
    "                evaluator_object = evaluator_validation_early_stopping,\n",
    "                lower_validations_allowed = 5, \n",
    "                validation_metric = \"MAP\")\n",
    "\n",
    "result_dict_funksvd, _ = evaluator_test.evaluateRecommender(recommender)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{5: {'ROC_AUC': 0.2677820121587237,\n",
       "  'PRECISION': 0.14230289832231663,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.14410258981742163,\n",
       "  'RECALL': 0.03642031146152129,\n",
       "  'MAP': 0.096818037826076,\n",
       "  'MRR': 0.28266116515519757,\n",
       "  'NDCG': 0.06774389907924616,\n",
       "  'F1': 0.05799712175093941,\n",
       "  'HIT_RATE': 0.7115144916116277,\n",
       "  'ARHR': 0.3678120985496398,\n",
       "  'NOVELTY': 0.004175950165821664,\n",
       "  'AVERAGE_POPULARITY': 0.7515882247201602,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.6512571033766005,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9302495545823865,\n",
       "  'COVERAGE_ITEM': 0.46587398183690665,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.021439940080516806,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.4445175877958728,\n",
       "  'DIVERSITY_GINI': 0.013815009439239335,\n",
       "  'SHANNON_ENTROPY': 4.847633844975068},\n",
       " 20: {'ROC_AUC': 0.44720211062555526,\n",
       "  'PRECISION': 0.06784982592875835,\n",
       "  'PRECISION_RECALL_MIN_DEN': 0.09772306205830338,\n",
       "  'RECALL': 0.07263442207382349,\n",
       "  'MAP': 0.04279839440348588,\n",
       "  'MRR': 0.30286159857056405,\n",
       "  'NDCG': 0.09491263757350027,\n",
       "  'F1': 0.07016064739235839,\n",
       "  'HIT_RATE': 1.3569965185747648,\n",
       "  'ARHR': 0.43414031830326105,\n",
       "  'NOVELTY': 0.02121106921879792,\n",
       "  'AVERAGE_POPULARITY': 0.4483648700630386,\n",
       "  'DIVERSITY_MEAN_INTER_LIST': 0.7361340418789873,\n",
       "  'DIVERSITY_HERFINDAHL': 0.9868061747697456,\n",
       "  'COVERAGE_ITEM': 0.9549667634116655,\n",
       "  'COVERAGE_ITEM_CORRECT': 0.09549667634116656,\n",
       "  'COVERAGE_USER': 0.9988694581985746,\n",
       "  'COVERAGE_USER_CORRECT': 0.6310712956867683,\n",
       "  'DIVERSITY_GINI': 0.1517651559828791,\n",
       "  'SHANNON_ENTROPY': 8.875972185000485}}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_dict_funksvd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What about the performance of these recommenders?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for MAP\n",
      "PureSVD\t|PureSVD-ItemBased\t|BPRMF\t|FunkSVD\n",
      "0.20\t|0.20\t|0.00\t|0.04\n"
     ]
    }
   ],
   "source": [
    "# MAP@20\n",
    "\n",
    "puresvd = result_dict_puresvd[20]['MAP']\n",
    "puresvd_i = result_dict_puresvditem[20]['MAP']\n",
    "bpr = result_dict_bpr[20]['MAP']\n",
    "funksvd = result_dict_funksvd[20]['MAP']\n",
    "\n",
    "print(\"Results for MAP\")\n",
    "print(\"PureSVD\\t|PureSVD-ItemBased\\t|BPRMF\\t|FunkSVD\")\n",
    "print(f\"{puresvd:.2f}\\t|{puresvd_i:.2f}\\t|{bpr:.2f}\\t|{funksvd:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for PRECISION\n",
      "PureSVD\t|PureSVD-ItemBased\t|BPRMF\t|FunkSVD\n",
      "0.24\t|0.24\t|0.00\t|0.07\n"
     ]
    }
   ],
   "source": [
    "# Precision@20\n",
    "\n",
    "puresvd = result_dict_puresvd[20]['PRECISION']\n",
    "puresvd_i = result_dict_puresvditem[20]['PRECISION']\n",
    "bpr = result_dict_bpr[20]['PRECISION']\n",
    "funksvd = result_dict_funksvd[20]['PRECISION']\n",
    "\n",
    "print(\"Results for PRECISION\")\n",
    "print(\"PureSVD\\t|PureSVD-ItemBased\\t|BPRMF\\t|FunkSVD\")\n",
    "print(f\"{puresvd:.2f}\\t|{puresvd_i:.2f}\\t|{bpr:.2f}\\t|{funksvd:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results for RECALL\n",
      "PureSVD\t|PureSVD-ItemBased\t|BPRMF\t|FunkSVD\n",
      "0.26\t|0.26\t|0.00\t|0.07\n"
     ]
    }
   ],
   "source": [
    "# Recall@20\n",
    "\n",
    "puresvd = result_dict_puresvd[20]['RECALL']\n",
    "puresvd_i = result_dict_puresvditem[20]['RECALL']\n",
    "bpr = result_dict_bpr[20]['RECALL']\n",
    "funksvd = result_dict_funksvd[20]['RECALL']\n",
    "\n",
    "print(\"Results for RECALL\")\n",
    "print(\"PureSVD\\t|PureSVD-ItemBased\\t|BPRMF\\t|FunkSVD\")\n",
    "print(f\"{puresvd:.2f}\\t|{puresvd_i:.2f}\\t|{bpr:.2f}\\t|{funksvd:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Extra\n",
    "\n",
    "This section is for you to practice and analyze different aspects of what you saw in the notebook.\n",
    "\n",
    "1. We briefly mentioned early stopping. Research on how to use it and when to use it. Implement it in models that might require it.\n",
    "\n",
    "2. The comparison we did was not fair in terms of parameters and their tuning. Run a hyperparameter tuning of the algorithms presented in the notebook and compare the best performant ones.\n",
    "\n",
    "3. Read about these other Matrix Factorization techniques:\n",
    " * Non Negative Matrix Factorization Recommender (NMFRecommender)\n",
    " * Binary/Implicit Alternating Least Squares (IALS) (IALSRecommender in the material)\n",
    " \n",
    " Familiarize yourself with these and compare them with what you've already know (BPR MF, FunkSVD, PureSVD, MF with PyTorch). What are the differences and similarities with what you've already seen?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
